We thank the reviewers for their thoughtful and constructive feedback on our manuscript. This should help both contextualize each task's difficulty and illustrate what it involves. Reviewer 3 noted the Section 2 task descriptions could be better presented. We have reformatted it so that "the order We also changed our description of IMP ALA to match Reviewer 5's suggestion. Regarding the task suite, Reviewer 4 raised a thoughtful consideration on whether "most of the findings translate when Some 3D tasks in the suite already have '2D-like' semi-counterparts that do not require navigation, '2D-like' because everything is fully observable and the agent has a first-person point of view from a fixed point, without Spot the Difference level, was overall harder than Change Detection for our ablation models.

Thank you to each reviewer for your helpful feedback on our paper. Below we provide our reasoning for several selected points. Due to page limits, only a portion of the updated figure is shown below. TrajGRU) would be out of scope (and well over page count). The baselines we provide show that depending on your choice of loss function, certain axes of "goodness" are brought We will add more discussion along these lines which address "what is done and why".